Repository navigation
Migrate remaining Ironwood training recipes from XPK to Cluster Toolkit - #392
Merged
ankitkinra merged 2 commits intoNov 15, 2026
Conversation
Migrates the 3 remaining Ironwood (tpu7x) pretraining recipes (MaxText and MaxDiffusion) from XPK to Cluster Toolkit (gcluster v1.108.0) with rendered Kubernetes JobSet manifests, removes their legacy XPK recipes, and updates training/ironwood/README.md benchmark metrics: - training/ironwood/llama3.1-70b/8k-bf16-tpu7x-4x8x8/ - training/ironwood/llama3.1-70b/8k-fp8-tpu7x-4x8x8/ - training/ironwood/wan2.2-t2v-14b/bf16-tpu7x-2x4x4/
There was a problem hiding this comment.
Code Review
This pull request migrates several training recipes on Ironwood TPU (including llama3.1-70b and wan2.2-t2v-14b) from XPK-based deployments to Cluster Toolkit (gcluster) deployments. This involves adding new Cluster Toolkit configuration files, run scripts, and updated Kubernetes manifests, while removing the old XPK-related files. Feedback on the changes highlights several instances of dead code in the newly added run_recipe.sh and run_workload.sh scripts for both the bf16 and fp8 llama3.1-70b recipes, specifically pointing out that the reinforcement learning (RL) conditional blocks always evaluate to false and should be removed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
As part of the XPK deprecation, this PR migrates the remaining three Ironwood (
tpu7x) training recipes from XPK to Cluster Toolkit (cluster_toolkit/andk8s/) and updatestraining/ironwood/README.mdwith hardware-validated benchmark metrics:training/ironwood/llama3.1-70b/8k-bf16-tpu7x-4x8x8/(12.48sstep time,1,179.62TFLOPs/s/chip,2,625.91tokens/s/chip)training/ironwood/llama3.1-70b/8k-fp8-tpu7x-4x8x8/(8.15sstep time,1,805.78TFLOPs/s/chip,4,019.79tokens/s/chip)training/ironwood/wan2.2-t2v-14b/bf16-tpu7x-2x4x4/(12.08sstep time,863.44TFLOPs/s/chip)For each recipe directory:
cluster_toolkit/(README.md,run_recipe.sh,run_workload.sh)k8s/k8s_manifest.yamlfrom the validated Cluster Toolkit (gcluster v1.108.0) execution